The one - loop vacuum energy and RG flow induced by double - trace operators in AdS / CFT and dS / CFT correspondence

We calculate the difference of one-loop vacuum energies for massive scalar field in five-dimensional AdS black hole. (The same is done in five-dimensional deSitter space). In each case this difference is specified by the boundary conditions corresponding to the double-trace operator (massive term) and it describes RG flow in the manner discussed by Gubser-Mitra for pure AdS space. For AdS black hole there occurs instability which is the manifestation of the Hawking-Page phase transition. For stable phase of AdS black hole as well as for deSitter bulk, c-function found beyond the leading order approximation shows the monotonic behaviour consistent with c-theorem.